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(54) INFORMATION MULTIPLEXER 
(57)Abstract: 

PROBLEM TO BE SOLVED: To provide an information multiplexer by which 
information codingtransmissiondecoding real time managementand synchronization 
multiplexing of individually coded information are easily conducted. 
SOLUTION: A coding parameter obtained by coding moving image data is given to a 
packet processing section 22 separately from coded data of input moving imageand 
the packet processing section 33 processes coded data into packets. Thusbased on 
the coding parameter 50time management information for decoding and reproducing 
coded data of a moving image is calculated and it is added to a packet of the coded 
data of the moving imagea multiplex processing section 43 applies synchronization 
multiplexing to voice coding data or the like and the result is outputted to a 
transmission line via an output buffer 45. 



CLAIMS 



[Claim(s)] 

[Claim 1]A multiplexing means which packet-izes coding data of videoand audio 
coding dataand multiplexes them at leastAn output means which outputs multiplexing 
data produced by multiplexing by this multiplexing meansProvide an input means 
which inputs an encoding parameter obtained when coding dynamic image dataand 
said multiplexing meansAn information multiplexing device computing time-of-day- 



control information on decoding and reproduction of coding data of said videoadding it 
to a packet of coding data of said videoand carrying out multiplex based on an 
encoding parameter inputted by said input means. 

[Claim 2]The information multiplexing device according to claim 1 wherein said 
encoding parameter contains a parameter of rearrangement to display order from 
decoding order of a coding frame of video. 

[Claim 3]The information multiplexing device according to claim 1 wherein said 
encoding parameter contains a parameter about a display period of a coding frame of 
video. 

[Claim 4]The information multiplexing device according to claim 1 carrying out 
multiplex [ of the encoding parameter inputted by said input means ] to coding data of 
said videoand inputting it into it. 

[Claim 5]Two or more inputted buffers which accumulate coding data of videoand 
audio coding data at least temporarilyrespectivelyA multiplexing means which packet- 
izes coding data accumulated in each of two or more of these buffersand multiplexes 
itProvide an output means which outputs multiplexing data produced by multiplexing 
by this multiplexing meansand said multiplexing meansAn information multiplexing 
device determining timing which starts multiplexing processing based on an 
occupation of said buffer of coding data in which transition of an occupation of said 
buffer can serve as time base among coding data accumulated in said buffer 
temporarily. 

[Claim 6]Two or more inputted buffers which accumulate coding data of videoand 
audio coding data at least temporarilyrespectivelyA multiplexing means which packet- 
izes coding data accumulated in each of two or more of these buffersand multiplexes 
itAn information multiplexing device determining timing which an output means which 
outputs multiplexing data produced by multiplexing by this multiplexing means is 
providedand said multiplexing means calculates a coding frame number by analyzing 
said coding dataand starts multiplexing processing based on these enumerated data. 



DETAILED DESCRIPTION 



[Detailed Description of the Invention] 
[0001] 

[Field of the Invention]This invention relates to the information multiplexing device 
which carries out packet multiplexing of the coded video bit stringthe coded voice bit 
stringand the encoded bit sequence of further othersand transmits. 
[0002] 

[Description of the Prior Art]MPEG-1 (ISO/IEC11172) which targeted the storage 
medium of the low bit rate comparatively as an international standard of the 
compression encoding system of videoMPEG-2 (ISO/IEC13818) etc. from which the 



application to next-generation digital broadcastingVOD (Video On Demand)or DVD 
(Digital Video Disk) etc. is expected is known. 

[0003]In the MPEG coding methodan imagea soundand other data streams (data row) 
are coded independentlythose coding data is packet-izedrespectivelyand the method 
which performs packet multiplexing is adopted. Between the transmitting side and a 
receivera synchronization is established by transmitting each time stamp of the 
receiver hitting time of a packetdecoding timeand display time by the frequency more 
than fixed. 

[0004]Each time stamp of decoding time and display time usually changes with 
encoding parameters of each picture and each sound. At the case where divide into 
the coding frame of a constant interval and coding is carried out to entry 
sequencedthese time stamps can be updated by always adding a fixed difference 
value. Howeverin MPEG video codingdisplay order may differ from encoding orderand 
the relation may change dynamically. In such a casein addition of constant valueit is 
necessary for renewal of a time stamp to become impossible and to perform suitable 
renewal of a time stamp from the header information etc. of the coding data inputted. 
[0005]In the transmitting sidewithout causing the breakdown of overflow of the input 
buffer in a receiverthe Anda flowetc.it must perform selection of the order of packet 
multiplexingand multiplex [ of a time stamp ] so that a synchronization may be 
established correctly. 

[0006]Usuallyin the multiplexing device which comprises hardwarethese time stamps 
are calculated based on the time base of a multiplex processing section. The time 
base of a multiplex processing section can operate to them and asynchronouswhen 
synchronizing with the clock of a transmission line or an input source. Usuallythe 
composition of a multiplexing system changes greatly with a 
synchronization/asynchronous one of these clocks. 
[0007] 

[Problem(s) to be Solved by the Invention]In an MPEG coding methodonly from the 
header information of the coding data inputtedin order to compute each time stamp of 
decoding time and display timeit must have delay of the maximum of the gap with 
display order and encoding orderand the header information of coding data must be 
analyzed one by one. In real timethis delaying amount cannot be disregarded in the 
transmission systems which perform codingtransmissionand decoding. 
[0008]In all the processings until it obtains the multiplexing bit stream [-izing / 
carried out multiplex / of them / and / coded individually the information from each 
sources of informationsuch as video and a soundand asked for the encoding data 
streamand / them / the bit stream / 1 ]In the information multiplexing device which 
assumed the synchronization to single time baseit becomes possible [ timethe number 
of encoded bitsand an encoding rate ] to connect 1 to 1 thoroughly. Thereforethe 
calculation of selection of the order of packet multiplexinga time stampetc. in a 
multiplexing device becomes possible [ calculating logically ] based on the virtual time 



base obtained from an encoding data stream. 

[0009]In such an information multiplexing devicerealization of multiplexing processing 
is attained only by software processingand the system construction of it which is low 
cost and is flexible becomes possible to the dedicated hardware for real time 
management. 

[0010]On the other handin the multiplexing processing by software processing without 
real time managementit is required that the throughput of multiplexing processing 
should be quicker than an actual code-ized rate enough at least. 

Thereforemultiplexing processing will be performed burstily and it will be accompanied 
by a big Jitter between the virtual time base and real time base on software. 
[001 1]Although it is usually necessary to absorb this Jitter by a transmission 
bufFerwhen performing coding and decryption using real time transmissionmaking the 
amount of buffer delay small as much as possible is called for. In a receiverit is not 
preferred to take the large input buffer delay more than needed for jitter absorption. 
Thereforewhen it assumes minimizing input buffer delay in a receiver and starting the 
usual decoding operation from the input time of an encoding data streamfor the gap 
with the virtual time base of a multiplexing partand real timeWhen the advance of the 
transmission start time of a multiplexing bit stream takes placea possibility of causing 
the underflow in a transmission line or a receiver arises. 
[0012]Thenthis invention is made in view of the above problemand is a thing. 
The purpose is to provide the information multiplexing device which can decrease in 
number the multiplexing delay produced for the analysis of header information by 
inputting the encoding parameter about the decryption time of other coding 
datadisplay timeetc. into a multiplexing part independently of the header information. 

[0013]The purpose of this invention is as follows. 

By presuming real time from the state of the input buffer of a multiplexing partand 
controlling a multiplexing bit stream transmission startperform real time management 
easily and prevent the above-mentioned underflow. 

Provide the information multiplexing device which does not need complicated 
hardware. 

[0014] 

[Means for Solving the ProblemjA multiplexing means which an information 
multiplexing device of this invention packet-izes coding data of videoand audio coding 
data at leastand is multiplexedAn output means which outputs multiplexing data 
produced by multiplexing by this multiplexing meansProvide an input means which 
inputs an encoding parameter obtained when coding dynamic image dataand said 
multiplexing meansBy computing time-of-day-control information on decoding and 
reproduction of coding data of said videoadding it to a packet of coding data of said 
videoand carrying out multiplex based on an encoding parameter inputted by said 



input meansin order to compute time-of-day-control information on decoding and 
reproduction of a time stamp etc.multiplexing delay produced in the analysis of header 
information can be decreased. 

[0015]Tv/o or more buffers into which an information multiplexing device of this 
invention was inputted and which accumulate coding data of videoand audio coding 
data at least temporarilyrespectivelyA multiplexing means which packet-izes coding 
data accumulated in each of two or more of these bufFersand multiplexes itProvide an 
output means which outputs multiplexing data produced by multiplexing by this 
multiplexing meansand said multiplexing meansBy determining timing which starts 
multiplexing processing based on an occupation of said buffer of coding data in which 
transition of an occupation of said buffer can serve as time base among coding data 
accumulated in said buffer temporarilyComplicated hardware is not neededwhile being 
able to perform real time management by software easily and preventing underflow in 
a receive buffer by the side of decoding. 

[0016]Two or more buffers into which an information multiplexing device of this 
invention was inputted and which accumulate coding data of videoand audio coding 
data at least temporarilyrespectivelyA multiplexing means which packet-izes coding 
data accumulated in each of two or more of these buffersand multiplexes itProvide an 
output means which outputs multiplexing data produced by multiplexing by this 
multiplexing meansand said multiplexing meansBy calculating a coding frame number 
and determining timing which starts multiplexing processing based on these 
enumerated data by analyzing said coding dataComplicated hardware is not 
neededwhile being able to perform real time management by software easily and 
preventing underflow in a receive buffer by the side of decoding. Thereforean 
information multiplexing device which can perform easily coding of 
informationtransmissionreal time management of decryptionand the synchronous 
multiplex one of information coded individually can be provided. 
[0017] 

[Embodiment of the Invention]One embodiment of this invention is described with 
reference to drawings. 

(A 1st embodiment) Drawing 1 is a block diagram showing roughly the composition of 
the information multiplexing device concerning a 1st embodiment of this invention. 
[0018] Drawing 1 shows the case where the dynamic image signal 10 of one 
channelthe audio signal of n channeland the data signal of one channel are multiplexed. 
It is inputted into the video coding equipment 15the voice to digital converter 16-1 - 
1 6-nrespectivelyand is codedand the audio signal 1 1-1 of two or more [ 10 or ] video 
(video) signals - 1 1-n are inputted into the input buffer 23and 24-1 - 24-n as 
individual coding datarespectively. 

[0019]Or it does not need codingthe already coded data stream 14 is similarly 
inputted into the input buffer 25. The data stored in each coding data and the input 
buffer 2f> which were accumulated in the input buffer 23and 24-1 - 24-n temporarily 



temporarilylt is individually packet-ized by the packetHzed treating part 3334-1 - 34- 
nand 35respectivelyand furtherthese packets multiplex by the multiplex processing 
section 43and a multiplexing bit stream is obtained. This multiplexing bit stream 44 is 
transmitted via a network according to predetermined timingfor exampleafter being 
accumulated in the output buffer 45 temporarily. 

[0020]The encoding parameter obtained when coding video and a sound by each 
coding equipment 15and 16-1 - 16-n inputs into video (video) and the packet-ized 
treating part 33 of an audio signaland 34-1 - 34-n as the side information 50-53. 
[0021 ]It is possible for the parameter in connection with the coding bit rateframe- 
coded-data lengtha displaying frame ratethe display period of each frameand the 
prediction structure of video codingetc. to be included in side information. 
[0022]In the packet-ized treating part 33and 34-1 - 34-npacket-izingcalculation of a 
time stampa decision of the order of multiplexingetc. are made using these side 
information. 

[0023]A part of these side information can also be extracted by conducting grammar 
analysis of the coding data inputted. Drawing 2 is for explaining the processing order 
and the row of the frame in MPEG video coding. 

[0024]The coding which combined the frame inner code-ized picture (I picture)the 
forward prediction coded image (P picture)and the both-directions prediction-coding 
picture (B picture) in MPEG video coding is possible. Howeverwhen B picture is 
includedencoding order differs from a display order. 

[0025]In drawing 2 it is an example at the time of using the prediction structure by 
which two B pictures always enter between I picture and P picture. In this casein the 
decryption sideafter carrying out three-frame lapse of period of I picture and the P 
picture from the decrypted timethey are displayedthey are memorized by the frame 
memory of a decoding device in the meantimeand are used for decoding processing as 
an image comparison of an estimated image. 

[0026]When multiplexing and transmitting video and a sound in the system multiplex 
standard of MPEGIt is multiplexed and transmitted to the header information to which 
the time (DTS:Decoding Time Stamp) which should be decodedand the time (PTS: 
PresentationTime Stamp) which should be displayed are specified by an MPEG 
system standard. 

[0027]In that of drawing 2 PTS of I picture and P picture adds and computes only the 
time of 3 frame periods to DTS of this picture. Obtained PTS can be used as DTS of I 
picture coded next or P picture. 

[0028]Howeveras shown in drawing 3 it can change arbitrarily andgenerally the number 
of B pictures inserted into I picture or P picture cannot compute PTS from DTS as 
mentioned above. Usuallywhen I picture or P picture coded by the next is inputted to 
a certain I picture or P picturethe number of B pictures contained between them is 
become final and conclusive. Thereforein the packetHzed treating part 33 of the video 
of drawing 1 1 or P picture cannot calculate a time stamp simultaneously with an 



inputand multiplexing delay until the coding data of I picture coded by the next or P 
picture is obtained will be produced. 

[0029]In the information multiplexing device in a 1st embodiment. By inputting the 
frame interval of I picture or P picture into the packetHzed treating part 33 as side 
information from the coding equipment 15A time stamp required for packet-izing is 
easily computableand packet-izing and multiplexing processing become 
possiblewithout producing the above-mentioned multiplexing delay at the same time 
the coding data of I pictures each or P picture is obtained. 

[0030]When coding considering the interval of I picture or P picture as always fixedit 
is also possible to input the constant value into the packet-ized treating part 33 for 
examplebefore an encoding start. 

[0031 ]In MPEG 2 video codingthe period which displays one frame coded as shown in 
drawing 4 can be arbitrarily set to the 2 field or the 3 fields. This is usedwhen carrying 
out frame rate conversion and displaying the sauce of a movie with a frame period of 
24 Hz with a television machine with a frame period of 30 Hz. In this casewhen the 
interval of I picture or P picture is coded as always fixedconstant value does not 
become butgenerally the interval of DTS and PTS of I pictures each or P picture 
produces multiplexing delay for the reason mentioned above. 

[0032]In the information multiplexing device in a 1st embodiment. As a field number in 
consideration of the period when a coding frame displays the frame interval of I 
picture or P pictureby inputting into the packet-ized treating part 33 as side 
information from the video coding equipment 1 5Packet-izing and multiplexing 
processing become possiblewithout producing the above-mentioned multiplexing delay. 
[0033]D rawing 7 is what showed an example of the data structure of MPEG video 
coding datalt comprises a sequence header in connection with the whole video 
sequencea GOP header in connection with the picture set (GOP:Group Of Pictures) 
which carried out grouping in the unit of about 15 pictures which starts with I picture 
and a header of each pictureand coding data. 

[0034]The coding data of composition of having been shown in drawing 7 is outputted 
from the video coding equipment 15and is inputted into the packet-ized treating part 
33 via the input buffer 23. While the data of composition as shown in drawing 7 is 
inputted into the packet-ized treating part 33 from the input buffer 23from the video 
coding equipment 15. The frame interval of I picture or P pictureOr the value 
expressed with the field number in consideration of the period when a coding frame 
displays the frame interval is inputted as side informationBased on this side 
informationa time stamp (DTSPTS) required for packet-izing is computedand if the 
data row of composition of having been shown in drawing 7 is packet-izedthe 
computed time stamp will be added and it will be outputted to the multiplex 
processing section 43. 

(A 2nd embodiment) Drawing 5 is a block diagram showing roughly the composition of 
the information multiplexing device concerning a 2nd embodiment of this invention. 



[0035]In drawing S identical codes are given to drawing 1 and identical partsand a 
different portion is explained. That isthe information multiplexing device shown in 
drawing 5 possesses the control section 60 which controls video and the audio coding 
equipment 1516-1 - 16-nand the packetHzed treating part 33and 34-1 - 34-n. The 
video shown by drawing 1 and the voice to digital converter 15and the side 
information acquired in the case of the coding by 16-1 - 16-n are inputted into the 
packet-ized treating part 33and 34-1 - 34-n via the control section 60, 
(A 3rd embodiment) Drawing 6 is a block diagram showing roughly the composition of 
the information multiplexing device concerning a 3rd embodiment of this invention. 
[0036]In drawing 6 identical codes are given to drawing 1 and identical partsand a 
different portion is explained. Namelythe information multiplexing device shown in 
drawing 6 multiplexes data required for especially calculation of a time stamp etc. in 
each coding data among the side information shown by a 1st embodimentand outputs 
them from the coding equipment 15and 16-1 - 16-n. 

[0037]It is also possible to multiplex an user datum after each headerfor example in 
the data structure of the MPEG video coding data shown in drawing 7 . The example of 
multiplexing of the user datum in that case is explained with reference to drawing 8 - 
drawing 10. 

[0038]In drawing S user data is multiplexed following a sequence header. The 
parameter in connection with the interval of the above-mentioned I picture or P 
picture is contained in this user datumand a time stamp is computed by a packet-ized 
treating part to it according to this parameter. The interval of I picture or P picture is 
constant through the sequence of one unitand when not using the above-mentioned 
frame rate conversionit is possible to reduce multiplexing delay using this technique. 
[0039]In drawing 9 user data is multiplexed following a GOP header. The parameter in 
connection with the interval of the above-mentioned I picture or P picture is 
contained in this user datumand a time stamp is computed by a packet-ized treating 
part to it according to this parameter. When the interval of I picture or P picture 
serves as variable and does not use the above-mentioned frame rate conversion for 
GOP unitsit is possible to reduce multiplexing delay using this technique. 
[0040]In drawing lO it is carrying out multiplex [ of the user data ] following the picture 
header. The parameter in connection with the interval of the above-mentioned I 
picture or P picture is contained in this user datumand a time stamp is computed by 
the packet-ized treating part 43 to it according to this parameter. In this caseit 
becomes always computable [ without multiplexing delay / a time stamp ] about 
arbitrary prediction structures. 

[004l]As mentioned abovein [ according to / as explained / a 3rd embodiment from 
the above 1 st ] a multiplex system [ need / time stampssuch as an MPEG system/ to 
be multiplexed ]The parameter which shows the relation of the entry sequenced and 
encoding order which are obtained when coding informationincluding videoa 
soundeto.by the coding equipment 15and 16-1 - 16-nthe parameter showing the 



display duration of a frameetc. are made into side informationlnput into the packet- 
ized treating part 33and 34-1 - 34-nand in the packetHzed treating part 33and 34-1 
- 34-n. It becomes possible to realize multiplexing processing in low delay as 
compared with the conventional method which computes a time stamp based on the 
information acquired by analyzing the header unit of coding data by what I will set 
calculation of a time stamp for one by one according to the inputted parameter. 
Usual lywhen computing a time stamp by analyzing coding datadelay of several frames 
arises theoreticallybut in this inventionthe theoretic delay for calculation of a time 
stamp is not produced. 

[0042]Although the side information containing an encoding parameter may be 
inputted from a course independent of coding datait can also simplify hardware by 
multiplexing to coding data beforehand and inputting into it. 

[0043]The encoding parameter can change dynamically per frameand that can be 
inputted into a packet-ized treating part one by oneand it can realize in it being fixed 
per sequence and inputting that into a packet-ized treating part only once to one 
sequence. 

[0044](A 4th embodiment) Drawing 1 1 is a block diagram showing roughly the 
composition of the information multiplexing device concerning a 4th embodiment of 
this invention. In drawing 1 1 identical codes are given to drawing 1 and identical 
partsand a different portion is explained. 

[0045]In drawing 1 1 multiplexing of a time stamp and control of the order of 
multiplexing are performed one by one according to virtual time base by the packet- 
ized treating part 3334-1 - 34-n35and the multiplex processing section 43. 
[0046]The control section 48 supervises the buffer occupied quantity of the coding 
data inputted into the input buffer 2324-1 - 24-nand 25and performs timing control of 
the transmission start of the multiplexing bit stream which notifies the time of the 
multiplexing processing of coding dataand the start of sending out to the multiplex 
processing section 43. For examplethe occupation of each input buffer notifies the 
timing used as the buffer occupied quantity equivalent to the time of the data transfer 
start timing logically drawn from the input buffer occupation of the audio coding data 
inputted with a fixed rate to the multiplex processing section 43. 

[0047]D rawing 12 shows more concretely the composition of the important section (it 
is the portion surrounded by the dotted line of drawing 1 1 and is hereafter called the 
multiplexing part 47) of the information multiplexing device of drawing 1 1 . The data 14 
of the coding equipment 15the video coded and outputted by 16-1 - 16-nthe audio 
coding data 1920-1 - 20-nand othersit is inputted into the coding data input device 
100101-1 - 101-nand 102respectivelyand further by DMA controller 110111-1 - 111- 
nand 1 12. Respectivelyit is transmitted to the input buffer 130 constituted on the 
main memory 125131-1 - 131-nand 132 via the data bus 120. 
[0048]Each transmitted data with CPU140 and the software 141 which manage the 
packet-ized treating part 33 of drawing 1 1 34-1 - 34-nand the processing operation 



equivalent to the function of 35 and the multiplex processing section 43. It is changed 
into a multiplexing bit stream and written in the output buffer 150 in the main memory 
125. 

[0049]The multiplexing bit stream written in the output buffer 150 is outputted to a 
transmission line by DMA controller 161 from the output unit 162 via the data bus 120. 
[0050]Here CPU140 and the multiplexing software 141 By counting the number of 
outputted bits of a multiplexing bit streamin a 5th embodiment that updates virtual 
time base and is mentioned laterthe coding frame number of each input coding data is 
countedand the processing which performs renewal of a time stampetc. is managed. 
CPU140 and the multiplexing software 141 constitute the control section 48 of 
drawing 1 land the control section 349 of below-mentioned drawing 18 . 
[0051]Dr awing 13 and drawing 14 are the figures for explaining the synchronization 
method which used the buffer. In drawing 13 coding data and 171 show a transmitting 
(output) buffer172 shows a transmission lineand 170 and 174 show the receiving 
(input) buffer 1 73. This transmission buffer and receive buffer are also called a 
smoothing buffer. 

[0052]The coding data 170 is inputted into the transmission buffer 171and after it is 
accumulated temporarily and delayedit is inputted into the receive buffer 173 via the 
transmission line 1 72. The transmitted coding data is outputted after delay is similarly 
added in the receive buffer 1 73. 

[0053]By controlling here so that the sum of the delaying amount in the transmission 
buffer 171 and the delaying amount in the receive buffer 173 becomes always 
fixedThe total delaying amount after the coding data 170 inputs into the transmission 
buffer 171 until it is outputted from the receive buffer 173 becomes always fixedand 
the data transfer which synchronized by transmission and reception becomes possible. 
[0054]timing chart ** of transmission and reception when the smoothing buffer of 
drawing 13 is used for drawing 14 — it carries out. In drawing 14 the image data of one 
frame of video is coded with a prescribed intervalthe coding data in which code 
amounts differ for every frame is generatedand it is sent to the transmission buffer 
171 one by one. 

[0055]190 shows the holding time in the transmission buffer 171 and 192 shows the 
holding time in the receive buffer 173. Although the holding time in the transmission 
buffer 171 and the holding time in the receive buffer 173 may be changed in time from 
the principle of a smoothing bufferrespectivelythe total delaying amount during the 
transmission and reception which are the sum is ♦*****(ed) so that it may become 
always fixed. 

[0056]In the coding mode with which the code amount of a frame unit is changed like 
MPEG video codingsmoothing of the amount of modulation codes which used the 
transceiver buffer 171 is performed. As mentioned aboveat the transmitting sideafter 
coding data is inputtedit is outputted after delay by the transmission buffer 171. Since 
each size of the transmission buffer 171 and the receive buffer 173 is the specified 



finite valuethe delaying amount in each buffer must be controlled strictly not to cause 
underflow and overflow with each buffer. 

[0057]D rawing 15 shows the input buffer occupation in the multiplex processing 
section 43 of each coding data of the transmitting side at the time of starting. Each 
coding data of a picture and a sound has encoding delay peculiar to a coding mode 
and an encoding parameteris inputted into the input buffer 130and 131-1 - 131-nand 
after passing through time to be equivalent to the further above-mentioned 
transmission buffer delayit multiplexes and it is sent out. 

[0058]In drawing 15 201 is coding data of a picture202 and 203 are audio coding 
datarespectivelyand the example in the case of multiplexing these three data is shown. 
By drawing 15a ll the coding data shows transition from the initial state of the input 
buffer occupation in the case of being inputted into the multiplexing part 47 at a fixed 
rate. Heresince the coding data of a picture has delay of the transmitting sideit is 
necessary to start transmission in the timing of the time 204and the audio coding 
data 202 and 203 shall start multiplexing and transmission from the timing of the time 
206 and 207respectively. 

[0059]Since the input interval of the frame which is a coding subjecta code amountan 
encoding rate which are produced by coding itetc. are not thoroughly connected with 
1 to 1 like the coding data of video when changing the code amount of a frame 
unitWhen software realizes the function of the multiplexing part 47management of real 
time (real time) becomes usually difficult only by performing smoothing by the above- 
mentioned buffer. 

[0060]That isif sending out is started by advance by the transmitting side (** which 
does not perform timing control of the transmission start of the multiplexing bit 
stream of this invention)a possibility of causing the underflow in a receiver for the gap 
with the virtual time base of the multiplexing part 47 and real time will arise. 
[0061]Thuswhen management of real time also takes into consideration the code 
amount of one frame (screen) which is one encoding unit in the case of the coding 
data of videolt can also be said to be the supervisory control of the transmit timing of 
the coding data in the transmitting side which holds a actual screen interval with the 
decoder of a receiverand can decode coding data. 

[0062]Thenthe input buffer occupation of a certain specific video or audio coding data 
202for exampleaudio coding data which is always inputted with a fixed bit rateis 
supervisedThe time of the data transfer start timing logically drawn from this flxed bit 
rate, (for examplein drawing 15t iming control of the transmission start of the 
multiplexing bit stream which starts multiplexing processing and sending out is 
performed from the time of becoming the buffer occupied quantity (for 
example drawing 15 input buffer occupation 205) equivalent to time 204). Therebythe 
underflow in a receive buffer which was mentioned above becomes possible 
[ protecting ]. 

[0063]Transition from the initial state of an input buffer occupation in case the coding 



data 210 of video is burstily inputted into the multiplexing part 47 in drawing 16 is 
shownand drawing 17 Transition from the initial state of an input buffer occupation 
while an input rate carries out the coding data 220 of video a time jitter to the 
multiplexing part 47in case it is inputted is shown. 

[0064]When an input is not a fixed rateit is difficult to presume real time from buffer 
occupied quantitybut. When there is attendant audio coded data (212 of drawing 
16222 of drawing 1 7 ) inputted with a fixed rateit is made to be the same as that of 
explanation of drawing IS From the input buffer occupation of the multiplexing part 47 
about the coding data. The buffer occupied quantity equivalent to the time (for 
example drawing 1 6 t he time 214 and drawing 1 7 t ime 224) of the data transfer start 
timing drawn logically, (for exampletiming control of the transmission start of the 
multiplexing bit stream which starts multiplexing processing and sending out is 
performed from the time of becoming the input buffer occupation 215 in drawing 
16and becoming input buffer occupation 225) in drawing 17 . 

[0065](A 5th embodiment) In the composition of the multiplexing part 47 shown in 
drawing 1 2since the input buffer 130 of each coding data131-1 - 131-nand 132 are 
constituted on the main memory 125the data will become transparent from software. 
Thereforeit is good also as composition which draws the current time for counting the 
coding frame number and controlling multiplexing start timing instead of measuring the 
occupation of the coding data in an input buffer. 

[0066] Drawing 18 is a block diagram showing roughly the composition of the 
information multiplexing device concerning a 5th embodiment. Drawing 18 shows the 
case where two or more programs which are the groups of video and a sound are 
coded and multiplexed simultaneously. Hereit is coded by the programs Chi and Ch2 
which comprise the video and sound which are coded in real timeand beforehandand 
carries out multiplex [ of the program of a total of three Ch(s) of program Ch3 
currently recorded on the storage medium ] simultaneously. The detailed composition 
of the important section (it is the portion surrounded by the dotted line of drawing 
18and is hereafter called the multiplexing part 347) of the information multiplexing 
device of drawing 18 is the same as that of drawing 12 . 

[0067]The video (video) signal SOOthe audio signal 301 and the dynamic image signal 
302and the voice picture 303It is inputted into the video coding equipment 315the 
voice to digital converter 316the video coding equipment 317and the voice to digital 
converter 318respectivelyis codedand is inputted into the input buffers 323324325and 
326 as individual coding datarespectively. 

[0068]The program data 304 read from the storage medium is similarly inputted into 
the input buffer 327. Or it does not need codingit may be made to input the already 
coded data stream 305 into the input buffer 351 similarly. 

[0069]Each coding data accumulated in each input buffer temporarily is individually 
packet-ized by the packet-ized treating parts 333-337and 

353respectivelyfurtherthese packets multiplex by the multiplex processing section 



343and a multiplexing bit stream is obtained. This multiplexing bit stream 344 is 
transmitted via a network according to predetermined timingfor exampleafter being 
accumulated in the output buffer 345 temporarily. 

[0070]In each of the packet-ized treating parts 333-337and 353the data analysis 
parts 333a334a335a336a337aand 353a which extract header unitssuch as coding data 
inputted from each input bufFerand are analyzed are provided. 

[0071]Based on the analysis result in these data analysis partsthe control section 349 
counts the coding frame number of videoand multiplexing start timing controli.e.real 
timeis computedand it notifies the timing which multiplexes and transmits to the 
multiplex processing section 343. About each program of Ch1-Ch3when the time base 
is independentthe virtual clock in multiplexing software is calculated 
independentlyrespectivelyand timing control of a multiplexing start is performed for 
every program. In a common programtime base shall also communalize the virtual 
clock in multiplexing softwareand shall perform timing control of a multiplexing start. 
[0072]As mentioned aboveas explainedwhen carrying out synchronous multiplex [ of 
coding data coded individuallysuch as video and a soundand the other data ] 
according to a 4th embodiment of the abovethe control section 48About the coding 
data of a specific picture or a sound which supervises the occupation of the data of 
each input buffer and is inputted with a fixed bit rate. The time of the data transfer 
start timing logically drawn from the fixed bit rate, (for examplein drawing 1 5t he time 
of becoming the buffer occupied quantity (for example drawing 15 input buffer 
occupation 205) equivalent to time 204) is notified to the multiplex processing section 
43and the multiplex processing section 43** which can realize multiplexing processing 
which can be equal also to real time transmission by starting multiplexing processing 
and sending out according to the timing notified from the control section 48 only by 
software By realizing the multiplexing part 47 only by softwareit becomes 
unnecessary to perform timing control by complicated hardwareand a system 
configuration can be simplified. Customization by software becomes easy and 
correspondence of it is flexibly attained to various applications. 
[0073]When carrying out synchronous multiplex [ of coding data coded 
individuallysuch as video and a soundand the other data ] according to a 5th 
embodiment of this inventionthe control section 349Based on the analysis result of 
the data analysis part of each packet-ized treating partcompute real time by counting 
the coding frame number of videonotify it to the multiplex processing section 343and 
the multiplex processing section 343** which can realize multiplexing processing 
which can be equal also to real time transmission by starting multiplexing processing 
and sending out according to the timing notified from the control section 349 only by 
software By realizing the multiplexing part 347 only by softwareit becomes 
unnecessary to perform timing control by complicated hardwareand a system 
configuration can be simplified. Customization by software becomes easy and 
correspondence of it is flexibly attained to various applications. It is effective even if 



it uses from the above Istcombining a 5th embodiment suitably. 
[0074] 

[Effect of the InventionlAs explained aboveaccording to this inventionthe encoding 
parameter about videoa soundthe decryption time of other coding dataand display 
time by inputting into a multiplexing part independently of the header informationThe 
information multiplexing device which can decrease in number the multiplexing delay 
produced in order to compute a time stamp by analyzing header information can be 
provided. 

[0075]The specific picture or input buffer occupation of audio coding data which is 
inputted with a fixed bit rate according to this inventionFrom the coding frame 
number in an input bufferor by controlling a multiplexing bit stream transmission start 
for them using the changed value to real timeWhile being able to perform real time 
management easily and preventing the underflow of the receive buffer by the side of a 
transmission line or decodingthe information multiplexing device which does not need 
complicated hardware can be provided. Thusaccording to the information multiplexing 
device of this inventionit can perform easily coding of informationtransmissionreal 
time management of decryptionand the synchronous multiplex one of the information 
coded individually. 



DESCRIPTION OF DRAWINGS 



[Brief Description of the Drawings] 

[Drawing 1] The block diagram showing the composition of the information multiplexing 
device concerning a 1st embodiment of this invention. 

[Drawing 2]T he figure for explaining the processing order and the row of the frame in 
MPEG video coding shows the case where the number of sheets of B picture is 
immobilization. 

[Drawing 3] The figure for explaining the processing order and the row of the 
concerned frame in MPEG video coding shows the case where the number of sheets 
of B picture changes. 

[Drawing 4] The figure for explaining the display period of each coding frame in MPRG 
video coding. 

[Drawing 5] The block diagram showing the composition of the information multiplexing 
device concerning a 2nd embodiment of this invention. 

[Drawing 6] The block diagram showing the composition of the information multiplexing 
device concerning a 3rd embodiment of this invention. 

[Drawing 7] The figure showing an example of the data structure of the MPEG video 
coding data in the information multiplexing device concerning the 1st embodiment and 
2nd embodiment. 

[Drawing 8] The figure showing an example of the data structure of the MPEG video 



coding data in the information multiplexing device concerning a 3rd embodiment. 
[Drawing 9] The figure showing other examples of the data structure of the MPEG 
video coding data in the information multiplexing device concerning a 3rd embodiment. 
[Drawing 10] The figure showing the example of further others of the data structure of 
the MPEG video coding data in the information multiplexing device concerning a 3rd 
embodiment. 

[Drawing 1 1] The block diagram showing the composition of the information 
multiplexing device concerning a 4th embodiment of this invention. 
[Drawing 12] The block diagram showing the composition of the important section of 
an information multiplexing device in details more. 

[Drawing 13] The figure for explaining the synchronization method using a transceiver 
buffer (smoothing buffer). 

[Drawing 14] The figure for explaining operation of a smoothing buffer. 
[Drawing 15]W ith the figure for explaining the timing control of the multiplexing 
processing and the transmission start concerning this inventionall the coding data 
shows transition from the initial state of the input buffer occupation in the case of 
being inputted into a multiplexing part at a fixed rate. 

[Drawing 16] The figure for explaining the timing control of the multiplexing processing 
and the transmission start concerning this invention shows transition from the initial 
state of an input buffer occupation in case the coding data of video is burstily 
inputted into a multiplexing part. 

[Drawing 1 7] Transition from the initial state of an input buffer occupation while an 
input rate carries out the coding data of video a time jitter to a multiplexing partin 
case it is inputted with the figure for explaining the timing control of the multiplexing 
processing and the transmission start concerning this invention is shown. 
[Drawing 18] The block diagram showing the composition of the information 
multiplexing device concerning a 5th embodiment of this invention. 
[Description of Notations] 

15 — Video coding equipment16-1 - 16-n — A voice to digital converter2324-1 - 
24-n25 [ — A multiplexing bit stream45 / — An output buffer47 / — A multiplexing 
part4860 / — A control section347 / — Multiplexing part. ] — An input buffer3334-1 
- 34-n35 — A packet-ized treating part43 — A multiplex processing section44 



